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ABSTRACT 

The  Joint  Integrated  Mission  Model  (JIMM)  is  a  real-time 
legacy  battlefield  simulator  employed  in  detailed  analyses 
and  virtual  exercises.  To  leverage  more  processors  to  im¬ 
prove  real-time  execution,  a  worker  pool  of  threads  opti¬ 
mistically  executes  events  in  parallel  but  avoids  cascading 
rollback  by  executing  only  one  future  event  per  simulated 
object.  Safeguards  for  maintenance  of  simulation  state  are 
programmed  explicitly  and  either  deferred  or  immediate 
modification  of  state  variables  could  be  employed  in  case 
of  event  rollback.  In  the  beginning  of  the  main  paralleliza¬ 
tion  effort,  deferred  modification  was  used  where  simula¬ 
tion  state  is  updated  only  when  the  event  can  be  completed 
safely.  However,  after  successful  implementation,  it  was 
determined  to  be  impractical.  Later,  all  safeguard  pro¬ 
gramming  employed  immediate  modification  where  origi¬ 
nal  state  is  restored  in  case  of  rollback.  This  paper  dis¬ 
cusses  these  techniques  for  parallel  execution  of  events  in 
JIMM  from  initial  efforts  through  later  code  maintenance. 

1  JOINT  INTEGRATED  MISSION  MODEL 

The  Joint  Integrated  Mission  Model  (JIMM)  is  a  general- 
purpose  (Nalepka,  Gump,  &  Kurker  2001)  real-time  dis¬ 
crete  event  simulator  primarily  used  for  forces  modeling 
and  simulation.  It  is  employed  as  the  main  threat  environ¬ 
ment  for  Naval  Air  Systems  Command  (NAVAIR)  Air 
Combat  Environment  Test  &  Evaluation  Facility 
(ACETEF)  test  and  training  exercises  (Mutschler  2007a). 
It  was  also  employed  for  requirements  generation  by  the 
Joint  Strike  Fighter  (JSF)  program  office.  Other  uses  in¬ 
clude  directed  energy  weapon  modeling  (Mutschler 
2007b),  weather  modeling  (Kelly  et  al.  2004),  communica¬ 
tion  modeling  (Chapman  and  Mutschler  2006),  radar  mod¬ 
eling  (Worsham  2002),  modeling  of  swarms  of  intelligence 
automata  (Niland  et  al.  2005),  air  defense  systems  (Du¬ 
quette,  Nalepka,  and  Luczak  2004),  and  human  behavior 
modeling  (Hoagland  et  al.  2001),  (Long  et  al.  2006). 


1.1  Parallelization 

JIMM  is  a  legacy  model  with  roots  extending  as  far  back 
as  1968  (JMMO  2008).  It  has  its  own  simulation  language 
to  allow  complex  interactions  as  well  as  a  graphical  user 
interface  for  quick  scenario  development  (Mutschler 
2005a).  However,  despite  its  already  efficient  operation, 
there  was  still  a  desire  to  leverage  multiple  processors  to 
execute  larger  and  more  complex  test  scenarios  in 
ACETEF  and  other  facilities  while  still  meeting  real-time 
deadlines.  Hence,  modifying  JIMM  to  employ  parallel 
processing  was  approved  and  funded  by  the  Common  High 
Performance  Computer  (HPC)  Software  Support  Initiative 
as  project  #7  of  Forces  Modeling  and  Simulation  (FMS  #7) 
Computational  Technology  Area  (CTA). 

1.2  Using  Worker  Pools 

JIMM  is  a  real-time  legacy  model  and  is  expected  to  oper¬ 
ate  in  both  single  processor  and  multi-processor  (high  per¬ 
formance  computing)  environments.  Therefore,  perform¬ 
ance  in  serial  operation  could  not  be  severely  impacted  by 
the  availability  of  parallel  operation.  In  addition,  shared 
memory  symmetric  multiprocessors  (SMPs)  were  very 
common  in  ACETEF  and  other  test  facilities. 

Hence,  threads  were  chosen  as  the  means  for  paralleli¬ 
zation.  They  permitted  separate  thread  processing  for  I/O 
and  event  execution.  This  would  improve  both  serial  and 
parallel  operation.  Threads  also  have  very  low  overhead 
and  can  be  used  in  the  SMP  environment. 

There  were  other  reasons  for  using  threads.  First, 
JIMM  is  a  product  currently  in  use  where  full  releases  are 
usually  provided  to  the  user  community  two  or  more  times 
per  year.  Use  of  threads  for  I/O  processing  could  be  pro¬ 
vided  to  the  community  earlier  as  a  more  immediate  bene¬ 
fit  of  the  parallel  operation.  Also,  there  was  a  significant 
fear  of  cancellation  of  the  parallelization  effort.  Providing 
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results  earlier  would  reduce  loss  should  that  cancellation 
occur. 

JIMM  is  also  used  in  real-time  environments  where 
steady  simulation  progress  is  the  major  requirement.  This 
was  thought  to  preclude  approaches  where  rollback  of  si¬ 
mulation  state  could  cause  intermediate  delays  in  simulator 
output.  In  addition,  events  in  JIMM  are  computationally 
small  and  there  was  a  major  concern  with  communication 
overhead,  even  in  a  shared  memoiy  threaded  processing 
environment.  Lastly,  there  was  a  desire  for  totally  repeat- 
able  operation  to  facilitate  analysis  and  allow  utilization  of 
the  extensive  test  programs  already  available  (Gibson  and 
Chapman  2001). 

Furthermore,  new  events  in  JIMM  can  be  scheduled 
for  the  same  simulation  time  as  their  parent  events.  This 
was  thought  to  preclude  conservative  parallelization  ap¬ 
proaches  where  a  minimum  non-zero  simulation  time  be¬ 
tween  an  event  and  successive  events  affecting  the  same 
simulation  object  is  required.  This  left  optimistic  ap¬ 
proaches  where  events  occurring  in  future  simulation  time 
are  calculated  assuming  little  chance  that  their  inputs 
would  change  (Fujimoto  1999).  However,  to  ensure  steady 
progress,  significant  rollback  common  to  some  optimistic 
approaches  had  to  be  avoided.  Hence,  future  processing  of 
parallel  events  was  limited  to  one  future  event  per  simula¬ 
tion  object. 

For  these  reasons  as  well,  a  general  worker  pool  ap¬ 
proach  (also  known  as  “scatter  and  gather  (SAG)”  or  “sin¬ 
gle  process  multiple  data  (SPMD)”)  was  used  instead  an 
approach  where  different  threads  execute  events  simulta¬ 
neously  and  communicate  simulation  state  through  mes¬ 
sages.  First,  the  worker  pool  approach  is  applicable  within 
the  SMP  environment.  Second,  overall  simulation  state  is 
saved  at  a  single  processing  point  once  an  event  is  finished 
processing.  Total  ordering  of  output,  regardless  of  the 
number  of  processors  can  then  be  achieved  permitting 
ready  use  of  available  test  capability.  There  is  no  commu¬ 
nication  overhead.  Lastly,  the  worker  pool  also  ensures 
that  one  event  currently  being  processed  is  the  earliest 
event  and  would  never  be  rolled  back,  thereby  ensuring 
minimum  forward  progress. 

Synchronization  overhead  was  still  a  major  concern. 
However,  events  in  JIMM  are  well  structured  and  affected 
simulation  objects  are  identified  before  event  execution. 
This  would  simplify  detection  for  the  need  for  rollback  af¬ 
ter  event  execution.  Also,  JIMM  scenarios  are  character¬ 
ized  by  a  large  number  of  simulated  entities.  This  would 
alleviate  overhead  from  rollback  processing  as  well  as  the 
limit  from  processing  only  one  future  event  per  simulation 
object.  Lastly,  significant  work  could  be  put  into  reducing 
processing  bottlenecks. 


1.3  Parallel  Operation 


The  general  architecture  divides  thread  execution  into  three 
parts:  upper  and  lower  critical  regions  that  are  protected 
by  a  single  common  mutual  exclusion  operation  (“mutex”) 
so  that  only  one  thread  can  execute  within  them  and  the 
last  part  where  multiple  threads  can  execute  events  in  par¬ 
allel  (Mutschler  2005b). 

After  simulation  start  and  until  simulation  end,  threads 
simultaneously  execute  in  a  loop.  After  simulation  start, 
the  threads  set  a  single  mutex  to  ensure  safe  serial  opera¬ 
tion  and  enter  the  upper  critical  region  (UCR)  where  they 
obtain  events.  They  exit  the  UCR,  unset  the  mutex,  and 
process  the  events  in  parallel.  After  processing  is  finished, 
the  threads  again  set  the  mutex  and  enter  the  lower  critical 
region  (LCR)  where  the  safety  of  final  event  processing  is 
determined.  New  events  are  queued.  Lastly,  output  and 
state  maintenance  are  arranged  for  later  processing  in  par¬ 
allel  before  executing  a  different  event.  The  threads  then 
enter  the  UCR  without  unsetting  or  resetting  the  mutex  and 
the  loop  begins  anew. 


File  Output 
External  Interfaces 
Displays 

State  Saving  or 
Rollback 


Figure  1:  JIMM  Architecture  with  Four  Threads 

In  the  UCR,  events  are  obtained  from  a  common  prior¬ 
ity  queue  ordered  by  simulation  time.  If  simulation  times 
are  equal,  then  a  unique  event  integer  identifier  that  is  in¬ 
cremented  and  assigned  when  events  are  initial  queued  is 
used  to  resolve  order.  A  Least  Global  Virtual  Time 
(LGVT)  is  defined  as  the  simulation  time  and  event  identi¬ 
fier  of  the  earliest  ordered  event  in  the  queue,  currently  be¬ 
ing  executed,  or  already  executed  but  awaiting  final  proc¬ 
essing  in  the  LCR  because  earlier  events  still  exist. 

After  the  event  is  obtained  from  the  event  queue,  it  is 
checked  for  possible  collisions  with  events  that  have  not 
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yet  finished  processing.  Event  collision  occurs  when  the 
event  examined  could  read  or  write  data  within  a  simula¬ 
tion  object  involved  with  a  later  event  (in  simulation  time) 
that  has  been  or  is  currently  being  processed.  Processing 
of  the  later  event  hence  might  no  longer  be  valid  because  it 
employed  data  that  might  have  changed. 

Should  a  collision  occur,  then  if  we  are  considering 
the  later  event,  it  is  queued  in  a  structure  on  the  executing 
event  and  rescheduled  once  that  event  completes.  Other¬ 
wise,  this  event  is  placed  on  a  different  list  on  the  execut¬ 
ing  event  and  the  executing  event  is  marked  for  rollback. 
Once  complete,  the  events  are  ordered  and  placed  back  on 
the  queue. 

If  no  collision  would  occur,  the  event  is  placed  on  a 
different  priority  queue  for  events  being  executed  or  events 
that  finished  executing  and  are  awaiting  final  processing  in 
the  LCR.  This  common  queue  simplifies  LGVT  calcula¬ 
tion. 

A  thread  would  execute  an  event  in  parallel  once  the 
mutex  for  the  critical  region  is  unset.  Changes  to  simula¬ 
tion  state  are  maintained  on  an  ordered  list  associated  with 
the  event.  Output  to  external  interfaces  is  also  kept  on  a 
separate  ordered  list  associated  with  the  event. 

The  thread  then  waits  upon  the  mutex  and  once  that  is 
satisfied,  enters  the  lower  critical  region  (LCR).  If  the  si¬ 
mulation  of  the  processed  event  is  equal  to  LGVT,  then  it 
is  processed  immediately.  Otherwise,  the  most  recent 
event  on  the  queue  of  executed  or  executing  events  is  then 
examined  to  determine  if  it  has  finished  executing  and  if  its 
event  time  would  be  the  LGVT.  If  so,  then  it  is  processed. 

Processing  of  an  event  with  LGVT  in  the  LCR  in¬ 
volves  several  other  actions.  First,  new  events  generated 
by  the  processed  events  are  queued  in  the  order  generated 
and  event  identifiers  assigned.  LGVT  is  checked  and  up¬ 
dated  when  the  new  event  would  be  the  earliest  unproc¬ 
essed  event  in  the  simulation.  Checks  for  collision  and 
subsequent  rollback  are  also  made. 

In  addition,  event  output  is  also  arranged  to  be  sent  in 
proper  order  outside  the  simulation  before  the  thread  proc¬ 
esses  its  next  event.  State  saving  (moving  forward)  or 
rollback  is  also  determined  to  be  executed  for  each  simula¬ 
tion  object  before  its  processing  of  its  next  associated 
event.  Event  output  and  simulation  object  rollbacks  are 
handled  in  the  main  body  and  outside  the  LCR  to  decrease 
the  impact  of  the  critical  regions  as  processing  bottlenecks. 

After  processing  an  event,  the  queue  of  executing  and 
executed  events  is  repeatedly  examined  and  the  earliest 
event  processed  until  the  earliest  event  time  is  no  longer 
the  LGVT. 

1.4  Optimizations 

A  number  of  optimizations  were  made  to  limit  the  impact 
of  the  UCR  and  LCR  bottleneck.  First,  these  regions  were 


coded  to  be  highly  efficient.  Aforementioned  optimiza¬ 
tions  include  the  following: 

•  Update  or  rollback  of  simulation  state  is  deter¬ 
mined  within  the  LCR  but  processed  outside  the 
critical  regions  before  the  next  pertinent  event  for 
the  affected  simulation  object. 

•  Submission  of  output  to  external  displays  and 
files  is  ordered  in  the  LCR  but  processed  before 
processing  of  a  thread’s  next  event. 

•  Use  of  a  common  queue  for  events  currently  exe¬ 
cuting  or  awaiting  final  processing  to  simplify 
LVGT  processing. 

Other  optimizations  include  the  following: 

•  A  common  memory  pool  already  employed  by 
JIMM  was  coded  for  parallel  operation  (Mut¬ 
schler  2006). 

•  Input  from  external  sources  is  also  handled  out¬ 
side  the  critical  regions. 

•  Each  event  is  assigned  a  main  simulation  object 
and  then  ordered  by  time  on  a  list  associated  with 
that  object.  This  list  is  referenced  as  a  single 
structure  on  the  main  simulation  queue  based  on 
the  time  of  its  earliest  event.  This  avoids  colli¬ 
sions  of  events  with  the  same  main  simulation  ob¬ 
ject. 

•  The  simulator  can  be  adjusted  to  obtain  more  than 
one  event  in  the  UCR  and  then  execute  them  one 
after  the  other.  This  can  reduce  synchronization 
overhead  due  to  mutex  operation  at  the  cost  of  po¬ 
tential  parallel  execution. 

•  A  separate  method  is  used  for  serial  execution  of 
events  as  opposed  to  event  execution  in  parallel. 
This  reduces  overhead  from  parallel  operation  ca¬ 
pability  when  it  is  not  necessary  and  there  no  ben¬ 
efit  from  parallel  operation  would  be  achieved. 

2  IMPLEMENTATION  OF  STATE  SAVING 

JIMM  is  implemented  in  the  C++  programming  language 
and  makes  extensive  use  of  object-oriented  programming 
constructs  such  as  derived  classes  that  inherit  data  and 
function  methods  from  associated  base  classes. 

Execution  of  events  in  parallel  was  implemented  in  the 
last  phases  of  the  parallelization  effort.  Associated  with 
these  phases  was  the  saving  of  the  state  of  objects  associ¬ 
ated  with  events  in  the  case  of  rollback. 

State  saving  data  is  implemented  as  a  doubly-linked 
list  of  objects  associated  with  a  single  simulation  object. 
The  objects  have  a  base  class  known  as  a  “result”.  Conse¬ 
quently,  the  list  is  known  as  a  “result  list”.  The  base  class 
includes  pointers  for  the  list  as  well  as  specification  of  two 
function  methods:  “Apply))”  where  the  simulation  state  of 
the  object  is  changed,  and  “Discard))”  where  the  simula¬ 
tion  object  is  returned  to  its  original  state.  The  derived 
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classes  of  the  result  base  class  contain  pertinent  data  as 
well  as  implementations  of  the  ApplyQ  and  Discard()  func¬ 
tions. 

Events  act  upon  simulation  objects.  At  the  start  of 
event,  the  result  list  for  each  of  the  simulation  objects 
should  be  empty.  When  the  event  is  processed,  results  are 
appended  to  the  result  list  as  needed.  When  an  event  is  de¬ 
termined  to  be  safe  (e.g.  has  the  earliest  LGVT)  or  rolled 
back  (e.g.  an  event  collision  has  occurred),  the  determina¬ 
tion  is  noted  in  a  bit  stored  within  the  simulation  object. 
The  results  are  later  processed  before  the  simulation  object 
is  accessed  in  a  different  event. 

If  the  event  is  determined  to  be  safe  for  final  process¬ 
ing,  then  the  result  list  is  processed  through  execution  of 
the  “Apply!)  "  function  from  earliest  result  to  last  result  to 
ensure  order  of  update.  If  the  event  must  be  rolled  back, 
then  the  result  list  is  processed  from  its  last  update  to  its 
earliest  through  execution  of  the  “Discard!)”  function  to 
ensure  correct  reverse  order  of  state  restoration. 

In  deferred  modification,  the  original  state  is  main¬ 
tained  in  the  simulation  object  and  changes  are  usually  (but 
not  always)  stored  in  the  result  class.  The  event  is  then 
coded  to  use  the  interim  values  and  ensure  that  the  initial 
state  in  the  object  is  not  altered.  Processing  of  the  result 
list  via  the  Apply!)  function  overwrites  the  initial  state  in 
the  simulation  object  with  the  modified  state.  Processing 
of  the  Discard!)  function  discards  the  changes  but  makes 
no  modification  to  the  original  simulation  object  state, 
leaving  it  intact. 

In  immediate  modification,  the  original  value  is  stored 
within  the  result  class.  The  data  in  the  original  data  struc¬ 
tures  for  events  are  then  modified  and  used.  When  the  Ap¬ 
ply!)  function  is  executed,  the  original  data  is  thrown  away 
since  saving  it  is  no  longer  necessary.  When  the  Discard!) 
function  is  executed,  the  original  state  is  restored. 

Saving  and  restoration  of  simulation  state  is  only  han¬ 
dled  by  result  classes  and  these  must  be  used  for  each  state 
change.  This  explicit  update  was  assumed  to  be  more  effi¬ 
cient  in  terms  of  performance  than  employing  classes  that 
perform  state  saving  automatically. 

Results  can  be  programmed  for  each  variable  type  as 
well  as  structures  or  other  collections  of  data.  In  the  be¬ 
ginning,  all  the  data  for  a  simulation  object  in  an  event  was 
handled  by  a  derived  result  class  instance.  Specific  code 
modules  were  denoted  by  a  “dr  ”  prefix  for  “derived  re¬ 
sult”. 

3  EVOLUTION  OF  STATE  SAVING 

As  a  legacy  model  in  current  use,  JIMM  is  updated  and  re¬ 
leased  two  or  more  times  per  year.  Most  corrections  and 
enhancements  are  achievable  as  single  stage  efforts  pro¬ 
vided  in  a  single  release.  Larger  efforts  are  divided  into 
increments  that  correspond  to  the  releases.  Because  work 


for  parallel  operation  was  a  multi-year  effort,  it  was  di¬ 
vided  into  parts  for  incorporation  into  successive  versions. 

The  early  part  of  the  parallelization  effort  from  August 
2000  to  2002  focused  on  use  of  threads  for  output,  allow¬ 
ing  multithreaded  access  to  terrain  and  the  memory  pool, 
better  organization  of  events  via  derived  classes,  organiz¬ 
ing  event  output  for  external  transmission  after  the  event 
completed,  and  eliminating  cases  where  simulation  state 
was  modified  outside  events  (JMMO  2008). 

Work  on  parallel  execution  of  events  started  in  late 
2001.  In  retrospect  however,  specific  development  and  in¬ 
tegration  of  parallel  execution  of  events  had  three  major 
stages:  initial  development  using  deferred  modification  up 
to  successful  demonstration  of  parallelism  starting  in  early 
2002  and  ending  in  2003,  rejection  of  deferred  modifica¬ 
tion  and  completion  of  the  development  effort,  and  subse¬ 
quent  perfective  maintenance  and  greater  use  of  generic 
classes  through  2007.  Employment  of  result  classes  has 
evolved  over  these  periods. 

3.1  Initial  Development 

When  the  parallelization  effort  was  initiated  in  2000,  the 
proposed  preliminary  architecture  was  examined.  Check¬ 
ing  of  collisions  in  the  UCR  was  not  thought  to  be  neces¬ 
sary  and  use  of  deferred  modification  of  simulation  objects 
would  allow  associated  events  to  be  executed  in  parallel 
safely.  This  assumption  was  later  rejected  since  update  of 
simulation  state  when  one  event  was  determined  to  be  safe 
could  still  occur  during  processing  in  the  main  body  of  a 
different  event  with  the  same  associated  simulation  object. 
However,  the  preference  for  deferred  modification  was  es¬ 
tablished. 

State  saving  using  deferred  modification  was  imple¬ 
mented  and  tested  for  several  small  events  and  shown  to 
operate  correctly.  Developers  initially  considered  the  ap¬ 
proach  to  be  straightforward. 

Generic  routines  for  saving  of  basic  types  such  as  in¬ 
tegers,  doubles,  pointers,  and  lists  where  implemented  but 
were  not  used  extensively.  Instead,  each  event  had  an  as¬ 
sociated  “result”  class  for  each  simulation  object  which 
handled  all  the  state  saving  processing. 

In  subsequent  development,  use  of  deferred  modifica¬ 
tion  worked  well.  Unfortunately,  as  more  complex  events 
were  considered,  retaining  and  using  the  intermediate  state 
instead  of  the  original  state  required  heroic  programming 
efforts.  Extensive  structures  containing  intermediate  data 
needed  to  be  retained  and  passed  from  procedure  to  proce¬ 
dure.  Coding  became  quite  cumbersome  especially  as 
common  procedures  and  classes  were  used  in  multiple 
events. 

Even  so,  the  methods  worked  for  an  initial  implemen¬ 
tation  involving  the  major  events  associated  with  sensing, 
perception  cognizance,  and  decision  making.  Near  linear 
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speedup  with  up  to  twenty-four  separate  processors  was 
obtained  (Mutschler  2005b). 

3.2  Rejection  of  Deferred  Modification 

Once  the  initial  parallel  version  was  complete,  many 
events  still  needed  to  be  coded  for  operation  in  parallel. 
This  effort  was  completed  by  the  JIMM  Model  Manage¬ 
ment  Office  (JMMO)  in  the  following  year  as  part  of  its 
code  maintenance  efforts. 

In  maintaining  the  code,  several  problems  became  ap¬ 
parent.  First,  event  implementations  could  rarely  mix  de¬ 
ferred  modification  (a.k.a.  “Apply-based”)  and  immediate 
modification  (a.k.a.  “Discard-based”)  methods.  An  event 
had  to  be  implemented  either  one  way  or  the  other.  This 
was  especially  a  cause  of  concern  given  the  use  of  common 
procedure  and  classes  and  the  now  inherent  utilization  of 
state  saving. 

Another  problem  was  that  much  of  processing  for 
events  employing  deferred  modification  was  moved  out¬ 
side  the  event  into  the  ApplyQ  function  of  the  result  class. 
This  made  the  code  difficult  to  follow  and  understand.  On 
the  other  hand,  events  employing  deferred  modification  re¬ 
tained  much  of  their  original  coding  inside  the  event,  thus 
making  understandability  and  subsequent  code  mainte¬ 
nance  much  easier. 

After  long  deliberation,  the  JMMO  determined  that 
even  though  many  events  were  already  implemented  using 
deferred  modification,  it  would  no  longer  be  used  and  that 
extensive  effort  would  be  undertaken  to  convert  events  im¬ 
plemented  using  deferred  modification  to  immediate  modi¬ 
fication  methods.  This  effort  has  been  completed.  How¬ 
ever,  instance  of  the  use  of  deferred  modification  are  still 
found  and  treated  as  low  priority  required  software 
changes. 

3.3  Using  of  Generic  Classes 

One  of  the  by-products  of  the  use  of  deferred  modification 
was  the  generation  of  many  instances  of  derived  “result” 
classes  specific  to  events  and  common  procedures.  The 
number  of  files  became  very  cumbersome. 

As  events  were  converted  (or  modified  to  operate  in 
parallel),  it  was  noted  that  many  functions  of  the  result 
classes  could  be  handled  by  several  instances  of  more  “ge¬ 
neric”  result  classes  that  operated  only  on  basic  data  types 
such  as  integers  and  pointers  or  on  simple  types  such  as 
lists.  Simulation  state  saving  would  not  be  done  in  a  single 
class  instance  but  would  be  done  with  multiple  smaller  re¬ 
sult  class  instances  on  the  event’s  associated  result  list. 

Because  the  generic  result  classes  only  dealt  with  sim¬ 
ple  types,  their  implementation  was  very  efficient.  Thus, 
the  associated  performance  cost  of  employing  them  vice  a 
single  result  class  instance  was  negligible.  Moreover,  code 


module  understandability  improved  significantly,  thereby 
reducing  overall  maintenance  costs. 

The  generic  result  classes  were  expanded  slightly  with 
instances  to  handle  simple  classes,  structures,  and  other 
blocks  of  data.  Otherwise,  they  were  left  unaltered. 

Eventually,  many  of  the  larger  event-specific  result 
class  instances  were  completely  replaced.  The  JMMO  then 
determined  that  all  event  or  procedure  specific  result  class 
instances  would  be  replaced  with  generic  results.  As  of 
November  2007,  less  than  a  dozen  of  these  instances  (from 
an  initial  count  of  more  than  one  hundred  fifty)  remained. 

The  JIMM  simulator  is  still  in  use.  The  release  of  ver¬ 
sion  3.2  was  provided  to  the  user  community  in  June,  2008 
(JMMO  2008).  Maintenance  of  parallel  execution  of 
events  continues. 

4  CONCLUSION 

This  paper  has  described  the  use  of  threaded  worker  pools 
to  execute  events  in  parallel  with  a  real-time  legacy  simu¬ 
lator  known  as  the  Joint  Integrated  Mission  Model 
(JIMM).  Methods  for  deferred  modification  and  immedi¬ 
ate  modification  of  simulation  state  variables  are  discussed 
and  shown  to  be  different  applications  of  a  common  state 
saving  class  known  as  “results”. 

During  initial  conversion  to  parallel  operation,  de¬ 
ferred  modification  techniques  were  employed  success¬ 
fully.  However,  as  the  code  was  later  maintained,  this  ap¬ 
proach  was  rejected  and  techniques  using  immediate 
modification  were  employed  instead  due  to  the  need  for 
increased  understandability  and  simpler  overall  construc¬ 
tion  given  complex  code  and  use  of  common  procedures. 

Eventually,  more  elaborate  derived  result  classes  spe¬ 
cific  to  events  and  common  procedure  were  replaced  with 
more  generic  result  class  instances  that  handled  basic 
types,  blocks,  and  lists. 
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